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ABSTRACT 

In 1992, eight two-year colleges in Northern 
California were chosen to participate in a quantitative study of the 
relationship between student outcomes and the use of instructional 
methods recomaended in the Cross/Angelo Classroom Assessment Model. 
The model supports the modification of instruction based on frequent, 
quick, and anonjnnous written feedback from students about their 
understanding of course content and their reactions to instruction. 
The study compared retention, grade distribution, and classroom 
environment data for classes in which Classroom Assessment techniques 
were used and those in which they were not employed. Outcomes 
comparisons were also made by gender and ethnicity. A first research 
design compared student retention for the same instructor before and 
after zhe instructor received training in and implemented Classroom 
Assessment techniques. In a second design, trained instructors either 
utilized or withheld Classroom Assessment techniques with different 
sections of the same course. Pilot studies were first conducted in 
summer 1992^ with larger studies conducted in fall 1992 and spring 
1993. Study findings i'ncluded the following: (1) retention rates for 
women in Classroom Assessment classes were higher than in control 
classes, though retention did not change for men; (2) grade point 
averages in Classroom Assessment classes were not significantly 
higher than in control classes, though there were a higher percentage 
of "A*s" among Classroom Assessment students; and (3) minority 
students felt more positive about the classroom environment in 
Classroom Assessment clasL*s than in control classes. A literature 
review; results of student and instructor surveys; recommendations; 
study letters and survey instruments; and a 52-item bibliography are 
included. (PAA) 
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CHAPTER ONE 



THE PROBLEM 



INTRODUCTION 



Since 1988, Drs. K. Patricia Cross and Thomas A. Angelo have been writing, teaching 
and speaking about the Classroom Assessment Model throughout the United States 
and Canada. This model, which Is based on the collection of frequent, quick and 
anonymous written feedback from students, has been embraced nationwide. 
Hundreds of instructors and thousands of students have bean exposed to the model 
and have supported the premise that when the instructor and student enter into a 
"partnership" for learning, positive and improved outcomes occur. Teachers feel 
better about teaching and students feel better about learning. Several qualitative 
studies {Stetson, 1989, Kelly, 19S2, Kelemen-Lohnas, 1993) have shown this to be 
so. 

It has been harder, however, to gather quantitative data to demonstrate statistically 
that there is a positive relationship between use of classroom assessment techniques 
and measures of student outcome, such as grades or retention. After initial funding 
for dissemination of the model throughout the state, the California Community College 
Chancellor's Office funded this attempt to measure the effect of the use of the 
Cross/Angelo model techniques on student outcomes. 
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BACKGROUND 



Eight California Community Colleges participated in this consortium research project. 
In 1991, the Chancellor's Office Grant supported dissemination training in the 
Cross/Angelo model at 17 Northern California colleges. In 1992, eight of these 
colleges were chosen to participate in the research study. Criteria used to choose the 
schools were continued school need for grant support to disseminate training, 
administrative commitment to the use of classroom assessment techniques by faculty 
in the classroom, and the perceived ability of the faculty leader to effectively train 
faculty and extend the model on each campus. The project director obtained signed 
agreements from college presidents and business office managers to accept 
Chancellor's Office funding in return for the training of faculty members at their 
colleges and the collection of research data about student outcomes in classes of 
those instructors who would be trained. Each participating college received $4,900 
to pay for a leader of the project, a research assistant, faculty participant stipends, 
a communications modem and a small travel budget. The project director and 
research coordinator met with the Deans of Instruction, administrator assigned to the 
project. Director of Admissions and Records, project leader and designated research 
assistant at each school. Agreements to participate were obtained. 

The research measures included comparison analysis of retention, grade distribution 
and classroom environment data for classes in which classroom assessment 
techniques were and were not used. There were two pilot testing trials, two small 
scale studies, and one large investigation conducted. It was anticipated that results 
and analysis of the data would assist colleges in the decision whether or not to fund 
classroom assessment programs by examining student outcomes. 

Participating schools in this consortium research project were: 

Napa Valley College 
Sierra College 
Cosumnes River College 
Chabot College 
Merced College 
Skyline College 
Butte College 
Gavilan College 
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TERMINOLOGY 



Clarification of common terminology in the Classroom Assessment field and in this 
document. 

CLASSROOM ASSESSMENT 

The process of using informal feedback techniques in which data is 
systematically gathered from students frequently and anonymously about their 
understanding of course content and reactions to instruction. Classroom 
Assessment could also include questions about student attitudes and 
background which may contribute to or impede their learning. 

CATS 

This abbreviation or acronym stands for "Classroom Assessment Techniques" 
or the actual small, continuous and anonymous assessment exercises done in 
the classroom. 

SUMMATIVE ASSESSMENT 

Often times called testing , summative assessment is usually done at the time 
instruction is completed or for purposes of grading. Summative assessment 
outcome, in the form of grades, is unchangeable. 

FORMATIVE ASSESSMENT 

Formative assessment is done while there is still time to change the outcome. 
When instructors use formative assessment prior to testing, there is still time 
for instructors to alter instruction to ensure that learning occurs. It "shapes" 
or "forms" learning while it is in progress. 

CLASSROOM RESEARCH 

Classroom Research is used to increase the teacher's understanding of the 
learning process. It differs from Classroom Assessment in that it usually tests 
a hypothesis or addresses a question-often one that arises from a Classroom 
Assessment. It capitalizes on teachers' knowledge of their disciplines and their 
students and does not require the knowledge of formal research design or 
statistical methods. 

CLASSROOM 

The place where an instructor meets and works with students and does 
assessments. This can be a classroom, clinical area or field placement. 

FEEDBACK 

The data gathered from students as a result of the assessment which is turned 
into helpful information for the instructor. 
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RETENTION 

In this study, those students who receive any grade (A, B, C, D, F, I or C/NC) 
on the final grading report are considered to be retained. This does not include 
those who dropped the course with a "W." The percentage generated between 
the nunnber of students enrolled at first census and those receiving grades is 
called retention, 

MATCHED CLASSES 

Matched classes were two sections of the same class taught by the same 
instructor at the same location at the same basic time. The two sections may 
have differed in year (pre and post instructor training in the model) or may have 
been taught simultaneously (after faculty were trained), depended upon the 
nature of the research design. 

FEEDBACK CLASS 

Class in which an instructor continuously, reliably and appropriately uses 
classroom assessment techniques. 

NON FEEDBACK CLASS 

Class in which an instructor withholds the use of any written and anonymous 
classroom assessment techniques. 

ADDITIONAL CONTROL GROUP 

A second control group added to the study in response to the pilot testing. 
This group included fifteen classes of students whose eleven instructors had 
never been trained in the Cross/Angelo method. 

CUCEI 

Tha College and University Classroom Environment Instrument developed by 
Drs. David Treagust and Barry Fraser at the Curtin Institute of Technology in 
Western Australia. 
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ASSUMPTIONS 



The Cross/Angelo Model of Classroom Assessment works well to increase 
student learning and student satisfaction with their learning experience. 

The Cross/Angelo Model of Classroom Assessment works well to increase 
instructor satisfaction with the teaching process. 
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OBJECTIVES 



The objective of this study was to examine the effect, if any, of classroom 
assessment on student outcome. It was an attempt to quantify student satisfaction 
and positive outcome which had been previously demonstrated mainly through 
qualitative measures. The questions asked were as follows: 

1. Does the use of Classroom Assessment Techniques have an effect on 
retention? 

2. Does the use of Classroom Assessment Techniques have an effect on 
final grades and grade distribution? 

3. Does the use of Classroom Assessment Techniques have an effect on 
course completion by gender? 

4. Does the use of Classroom Assessment Techniques have an effect on 
course completion by ethnicity? 

5. Does the use of Classroom Assessment Techniques influence the 
classroom environment? 



NULL HYPOTHESIS 

The null hypothesis was that there would be no differences in 
measurements of student outcome (retention, grade point average and 
grade distribution, course completion by gender/ethnicity and the 
classroom environment) in courses where Classroom Assessment 
Techniques were used when compared to matched classes where 
Classroom Assessment Techniques were not used. 
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SCOPE AND LIMITATIONS 

It is difficult in the community college system to analyze why some students persist 
and succeed and others drop out or fail. Socioeconomic level, family, and personal 
concerns certainly influence these outcomes. In this study, however, we examined 
only the effect of a certain teaching technique on outcome variables, and undoubtedly 
there were other variables influencing a student's success or non-success. This 
limitation was addressed by matching the control groups to the treatment groups as 
closely as possible, anticipating that extraneous variables influencing one group would 
influence the other group equally, and therefore be balanced out. 

This study was an examination of student outcome in response to instructor use of 
a particular teaching model. Upon piloting our research methodology, several 
questions arose. Could we be sure that instructors in the study were correctly trained 
in use of the model? Could we depend upon the fact that they were using the 
teaching techniques continuously, reliably and appropriately? Could we be sure that 
the techniques were being withheld in the control group classes, and that there was 
no crossover of information from the CATs classes to the control group classes? As 
time passed, we learned to be clearer and clearer in our directions to participating 
instructors. Uncertainty about instructor compliance to the research design was an 
initial limitation, and certainly influenced portions of the CUCEI classroom environment 
instrument pilot test results. 

An unanticipated serious limitation was instructor reluctance to withhold the 
Classroom Assessment Techniques in their classrooms. So convinced were 
participants in the effectiveness of the model that in several cases instructors wsre 
unwilling to finish the study due to ethical concerns. This prevented the inclusion of 
all eight schools into every portion of the study. 

Another limitation of the study was frequent difficulty in obtaining enrollment and 
completion data from the admissions and records departments at the participating 
colleges. The lack of timely and easy access to data by means of computer call up 
attests to the limitation that exists in California in 1993 with no easy data base 
access to student records in many community colleges. At some schools records had 
to be hand pulled and data taken from instructor roster due either to a very long time 
lag before records such as final grades were entered into the college information 
system or unanticipated early deletion of first census data entry into the system. 

The inclusion of one college that operated on the quarter system with seven colleges 
on the semester system proved to be a greater problem than originally foreset.i. That 
school's data was, in some instances, not able to be included in the analysis due to 
project schedule time constraints. 

The size of the study was limited by the funding available. Originally writtento study 
seventeen schools in both Northern and Southern California, when funded, financial 
support was sufficient to include only eight schools, all located in Northern California. 
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An attempt was made to include as diverse a population as possible, choosing both 
rural and urban schools with both large and small student populations. Six to eight 
instructors from each school participated in the research project. It is possible that 
the results obtained might be similar throughout other schools in the California 
Community College system. 
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WHAT IS THE CROSS/ANGELO MODEL? 

Classroom Assessment, as developed by Drs. K. Patricia Cross and Thomas A. 
Angelo, is a procedure which consists of doing small-scale assessments continuously 
(and anonymously) in college classrooms by discipline based teachers to determine 
what students are learning in that class (Cross, 1989). By using this method, the 
traditional "instructor in rront of the class" lecture process changes to a dynamic 
partnership where both parties direct the pathway to knowledge acquisition. Initially, 
faculty are prone to confuse Classroom Assessment Techniques (CATs) with the tests 
and quizzes they use to evaluate student learning. Unlike tests and quizzes, however, 
CATs are ungraded and anonymous. The purpose of doing CATs is to quickly assess 
learning in order to adjust instruction, not to evaluate achievement. CATs are usually 
done on 3X5 cards which are passed out either before, during or after class, on which 
students answer instructor generated questions. The primary purpose of Classroom 
Assessment is to improve learning directly, by providing teachers with the kind of 
feedback they need to make informed instructional decisions (Angeio, 1991). 

Despite the fact that all good teachers elicit feedback and use the responses 
informally, very few do so systematically and regularly enough to use that feedback 
to help students improve the quality of their learning. First, in informal data gathering, 
only a brave few may be willing to give an instructor feedback. Shy students, those 
whose culture teaches that questioning authority is undesirable or those students 
whose mastery of the spoken language is limited, would be hesitant to give oral or 
public feedback. 

Classroom Assessment Techniques offer faculty the opportunity to discover not just 
whether students are learning, or what they are learning, but how they learn and how 
well they learn in respor*se to how we teach, and they give faculty the opportunity 
to find these things out in time to help students immediately (Kort, 1991). Although 
many college faculty are well-trained experts in their fields of study, they often have 
little or no formal training or experience in systematically studying studen.C learning 
(Angeio, 1991). What keeps many (instructors) from being more effective teachers 
is not a lack of wanting but of not knowing quite how to do it (Katz, 1987). 
Classroom Assessment is an effective assessment model because it occurs as close 
as possible to the scene of action in teaching and learning (the classroom) and 
provides diagnostic feedback to both teachers and students-to teachers on how they 
can improve their teaching, to students on how they can improve their learning 
(Cross, 1988). 

The Classroom Assessment Model as developed by Cross and Angeio has been 
presented, studied and taught to faculty and administrators throughout the United 
States and Canada (Cross and Angeio, 1 993). Faculty and student acceptance of the 
model was immediate. People began to write about the use of the Classroom 
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Assessment Techniques as developed by Cross and Angelo and to conduct research 
studies on the application and results of applying the model. This research study 
attempts to examine the effects of the Cross/Angelo model on student outcomes by 
examining areas previously researched, such as retention and grades, and by 
introducing another area of research, that of the classroom environment. A literature 
review of related research on classroom assessment and classroom environment will 
follow. 



CLASSROOM ASSESSMENT RESEARCH 

The Classroom Assessment Model has roots that may be traced to a number of 
learning theories. For purposes of this research, two conceptual frameworks were 
used. One was Bloom's model of alterable classroom variables (1980), and the 
second was Lewin's (1936) field theory. Bloom's work with the notion of formative 
assessment provides a backdrop to the work of Cross and Angelo. Bloom (1980) 
states" Formative tests are used primarily as feedback, to inform the student about 
what he has learned well and what he still needs to learn. When feedback is provided 
in relation to corrective procedures to help the student correct the learning, then with 
additional time and help most students do reach the standard of achievement set by 
the teacher." 

Wilson (1986) was a researcher who also reported some of the first work done with 
feedback in the classroom. Wilson reported that a physics professor who was known 
as a "good instructor" wanted to know answers to two questions at the end of each 
lecture. Basically, they were: What the student thought the key point of a lecture 
was? And, what the student wanted to know more about? He planned to ask 
students these questions 4 or 5 times during the course of the semester. 

Cross, in working with faculty from a variety of New England colleges and 
universities, cited Wilson's work as an example of student feedback and interested 
one faculty member in Implementing an early version of what today is called 
Classroom Assessment Techniques (Mosteller, 1989). 

Mosteller (1989), attracted to the notion of student feedback, began to ask students 
to respond to one question at the end of each lecture. He asked: "What key point 
did the lecture cover?" He later added a second question to the feedback: "What 
then would you like to know more about?" He summarized the feedback, did 
frequency distributions, distributed them to his students, then asked his students how 
they liked the feedback process. Student responses indicated that the process "gave 
them a quick opportunity to integrate the lecture at the end of the hour" and they 
regarded this as beneficial. However, they added that the professor wasn't really 
getting the feedback that he needed about lingering student questions as he had 
already supplied students with the key points of the lecture. Thus, as a result of this 
feedback, the "muddiest point" feedback mechanism came into existence. Mosteller 
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changed his classroom question to ask students for the "muddiest point" in the 
lecture, and the feedback from this question led to what students still needed to know 
more about. 

Mosteller's study is considered a precursor of the Cross-Angelo model. It was 
developed using the notion of formative assessment by instructors to effect outcomes 
in their classrooms. The model suggests that when instructors have a means to 
assess how students are doing in a class prior to a test situation, the data obtained 
allows instructors to alter instruction to insure that learning occurs. This type of 
assessment is called formative assessment as it is done while there is still time to 
change the outcome. Summative assessment, oftentimes called a test, is usually 
done at the time instruction is completed for a particular unit and the instructional 
outcome for the material may not be changed. 

Buckheister (1989) looked at institutional effectiveness issues for faculty in relation 
to the Cross-Angelo Classroom Assessment Model. She stated that this was a 
method that could be employed to improve student learning as "the quality of learning 
depends directly on the quality of teaching...," and that student feedback should 
improve the quality of teaching if the faculty member were open to critique. 

Angelo (1990) suggests that classroom assessment is part of the larger arena known 
as classroom research. He states that if the quality of higher education is to improve, 
college faculty will need to become better skilled at assessing student learning in their 
own classroom and suggests that classroom assessment and research following 
feedback from classroom assessment is one way for faculty to do this. 

Cuevas (1991) surveyed faculty and administration who had participated in a 
classroom feedback course based on the Cross-Angelo model at Miami Dade 
Community College in conjunction with the University of Miami. He found that a 
random sample of those who participated in the course felt the model would be a 
beneficial one to use in a classroom situation. He also suggests that classroom 
assessment allows instructors to align what they teach with what they are assessing 
(Cohen, 1987). Given feedback from students, instructors, in follow-up sessions, can 
match what they present with what students say they need. Cuevas (1991) suggests 
that faculty need to be taught: How to use Classroom Assessment Techniques; how 
to modify instruction to match feedback; how to teach the necessary study/learning 
techniques students need to use; the difference between formative and summative 
assessment; and, that the institution needs to support faculty endeavors in this arena 
by acknowledging their efforts with either stipends or release time. 

Civikly (1992) looks at the issue of the relationship between teacher clarity and 
student outcomes, and states that there is a positive relationship between the two. 
Students in his study felt that the clearer an instructor was, the more positive was 
their perceptions of teaching effectiveness. "Teacher clarity was correlated positively 
with college students' positive judgments about their classes, attitudes about the 
instructor, and willingness to engage in the behaviors taught in class (Civikly, 1 992)." 
This research supports Cross and Angelo's work, in which student feedback leads the 
instructor to which topics taught still need clarification. 



In the past few years, researchers who believe in the Cross/Angelo model's 
effectiveness have tried to examine the model's effects on student outcomes. Using 
instructor application of CATs as the independent variable, the dependent variables 
most often studied have been student retention and grades. As part of program 
review at the College of Marin in California, the school examined retention and student 
grades in relation to instructor use of the model. College of Marin reported that they 
could not substantiate grade point differences in the use of the model, nor increases 
in retention. What was found was that faculty felt the process had improved 
instruction, had improved their relationships with students, and had improved their 
relationships with other faculty (Office of Staff and Organizational Development, 
College of Marin, 1990). 

The concept of instructor satisfaction with the model was also a major finding in the 
report of Eve Keleman-Lohnas in the FIPSE Grant Report (1993). Keleman-Lohnas 
reviewed hundreds of instructor summaries after training and use of the Cross/Angelo 
model. Instructors consistently reported "change in their teaching techniques, a new 
insight into the teaching/learning process, and increased professional growth." 
Instructors felt that their students became better listeners and learners who came 
better prepared for class. Faculty also reported that using the goal matching exercise 
led to improved student placement and increased student refcjrrals to appropriate 
campus resources. Instructors were satisfied with using CATs in the classroom; they 
felt much closer with students and they felt especially positive about the opportunity 
to sit and talk with fellow faculty about teaching as part of the training (Keleman- 
Lohnas, 1993). 

in a study done by Obler, Arnold, Sigala, and Umbdenstock (1 991 ), it was found that 
faculty universally commented that the best part of the classroom assessment training 
was the opportunity to talk with their colleagues. They cite the use of CATs by 
faculty as "recharging the batteries" of even the most experienced teachers. These 
researchers found that students, too, expressed satisfaction with the CATs process. 

Kelemen-Lohnas also collected data from students. In her study, students described 
increases in vation, sense of participation, satisfaction, and enjoyment of learning. 
They described an increase in student-faculty rapport, and satisfaction with the 
partnership process (Kelemen-Lohnas, 1993). 

Stetson (1 989) tried to demonstrate quantifiable effects of the model using retention. 
She was able to demonstrate a slight effect on retention. This slight positive effect 
on retention was further substantiated by Obler, Arnold, Sigala, and Umbdenstock 
(1991) with a diverse student population. Kelly (1992) looked at use of CATs 
feedback effect on retention for adult students, and she, too, found slight effect on 
retention. 

In a recent speech given by Thomas A. Angelo (June, 1993), he reports on general 
trends in research findings that examine the effect of classroom assessment on 
faculty and learners. These were: 
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1. There seems to be a change in faculty behavior. 

2. Student retention either stays the same or is slightly increased. 

3. Classroom assessment does not seem to harm students, and in some cases, 
classroom assessment may improve what is going on in the classroom. 

Because both students and instructors have often been cited as enjoying the use of 
the model in the classroom, literature was then reviewed that examines classroom 
environment as it relates to classroom assessment. 



CLASSROOM ENVIRONMENT RESEARCH 

The literature was reviewed to look for instruments that might be used to measure 
classroom environment. The work of Treagust and Fraser (1986) was found. These 
professors created The College and University Environment Inventory (CUCEI), an 
instrument which was developed after an extensive review of all contriouting literature 
regarding classroom environment and it's influence on student outcomes. 

Treagust and Fraser cited the influence of Chavez's work on the development of their 
inventory. Chavez, in 1 984, did a meta-analysis of all studies which connect different 
type of leadership roles and the "concomitant group climates." He cites a study by 
Lippitt in 1940, with an adolescent male population, in which Lippitt concluded that 
"different leadership styles produced different social climates and resulted in different 
group and individual behaviors." Chavez cites other studies in the 1 940's (Anderson 
and Brewer, 1945, and Anderson, Brewer, and Reed, 1946) which then extended this 
research to include the effects of teachers' personalities on students' behavior. A 
significant contribution made by these researchers was the identification of two 
classes of teacher behaviors: 1) "socially integrative behavior which promoted 'the 
interplay of differences' in a classroom and permitted original thinking to occur" and 
2) "socially dominative behavior which was characterized by a rigidity or inflexibility 
of purpose, an unwillingness to admit the contribution of another's experience, 
desires, purposes, or judgment." Chavez cites Withall (1949), who found that 
different teachers with the same students could change the classroom climate. 

Chavez also relates the development in the 1960's of an instrument known as ihe 
Observation Schedule and Record, the OScAR (Medley and Mitzel). This instrument 
had 14 response keys and 3 scales (emotional climate, verbal emphasis, and social 
organization). The research conducted by Medley and Mitzel using the OScAR found 
that a classroom's emotional climate was an important dimension in classroom 
research. 

Chavez cites several other instruments that were developed over the years to measure 
classroom environment. These included the College Characteristics Index (CCD, the 
High School Characteristics Index (HSCI), the Evening College Characteristics Index 
(ECCI), and the Organizational Climate Index (OCI). All four were similar in that they 
asked students to respond to true and false questions about aspects of environment 
in colleges, high schools, and evening college settings. 
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Treagust and Fraser attribute the development of their work to that of Trickett and 
Moos (1970), who developed a Classroom Environment Scale (CES) that looked at 
nine scales each containing 10 questions each. The nine scales were involvement, 
affiliation, support, task orientation, competition, order and organization, rule clarity, 
teacher control, and innovation. They found that these scales were able to reliably 
measure different types of interactions that occur in classroom setting. In 1973, a 
further study by Trickett and Moos found that the CES was able to discriminate 
between environmental factors that were different in different classrooms: 

Moos and Moos (1978) used the CES to further investigate the relationships among 
the following variables: classroom environments, grades, and absenteeism and found 
that there was a relationship between a stringent grading policy and increased student 
absences. There were also strong relationships between student and teacher 
perceptions of class environments and mean grades. They concluded that a 
competitive environment may have a positive effect on student grades but resuii: in 
more absences for a large group of students. Additionally, since absenteeism is 
related to poorer grades and subsequent drops, a competitive environment may 
benefit a few students at the expense of other students. 

Concurrent with the work described above, two other inventories were developed, the 
Learning Environment Inventory (LEI) and the My Class Inventory (MCI). The MCI was 
used with elementary and junior high school students (Chavez, 1984). 

The Learning Environment Inventory (LEI) was developed in the late sixties by 
Walberg. The LEI was originally designed with 14 scales and then added a fifteenth 
scale. These scales are: cohesiveness, diversity, formality, speed, environment, 
friction, goal direction, favoritism, cliqueness, satisfaction, disorganization, difficulty, 
apathy, democraticness, and competitiveness. 

The various studies done with the LEI (Walberg and Anderson, 1968; Walberg, 
1969a; Walberg, 1969b; Anderson et.al, 1969; O'Reilly, 1975; Randhawa & 
Michayluk, 1975) all found that a relationship existed between classroom 
environments and student outcomes. 

Fraser (1978) used a modified version of the LEI with junior high school students. He 
found that the inventory could indicate differences in student perception about the 
classroom environment. It could also predict student outcomes in both the affective 
and cognitive domains. 

The College and University Classroom Environment Inventory (CUCEI) was an 
outgrowth of the LCI and was developed to measure the environments in college 
seminar classes. The CUCEI evaluates students' perceptions of seven psychological 
dimensions of actual or preferred classroom environments. The scales are: 
Personalization, involvement. Student Cohesiveness, Satisfaction, Task Orientation, 
Innovation, and Individualization. The CUCEI was found to be a reliable measure of 
classroom environment both in four-year institutions and alternative high schools 
(Williamson et al., 1986) in Australia. 
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Research findings using this instrument suggest that classroom environments have an 
effect on student outcomes. Treagust and Fraser (1986) found that there was a 
strong relationship between classroom environment and student learning as measured 
by post-test scores. 

Bloom (1 980) reported new methods of evaluating what is going on in classrooms and 
suggested that classroom environments play a role in student outcomes. He states 
that the "qualities of teaching that have a direct causal relation with student learning 
in the classroom." 

Cross (1990) and Angelo (1990) suggest that use of Classroom Assessment 
Techniques is a methodology which gives instructors information about student 
perceptions of the classroom environment. Dependent upon classroom feedback 
techniques used, instructors can informally assess how students feel about specific 
cognitive and affective issues. 

One consistent outcome of classroom environment studies is the relationship between 
students' sense of cohesiveness and retention and achievement in the classroom. Uri 
Treisman's 1985 work documented the effects of group cohesiveness on student 
outcomes. Triesman saw that Black students in Math 1 A classes were not doing as 
well as other minority students, in particular Asian students. Through a 
comprehensive study of the two groups, which included researchers going home and 
living with the students (Triesman, 1992), he was able to isolate one variable that 
seemed to differentiate between the two groups. Asian students tended to study 
together in a group and contribute to each other's success in the class. Black 
students tended to study by themselves, and when confronted by a problem that 
seemed unsolvable, had no one to go to for help; leading to unfinished or incorrect 
homework, a sense of isolation, poor self image, and an eventual drop out or failure 
of the class. While the study groups served the Asian population both academically 
and socially, the Black students did not benefit from this type of support. 

As a result of these findings, Treisman began his Study Group Project, which resulted 
in Black students doing as well as or better than the rest of the class. Effect sizes (for 
final grade earned) for black students f articipating in the workshops were 1.18 in the 
1982 school year and .916 in 1983 when compared to Black students not in study 
groups. Triesman's project of study groups for minority students had three goals: 

• Achieve mastery of the work; 

• Identify early problems and work with students to alleviate problems before 
they became insurmountable; and 

• Help students develop skills and resources that will enable them to excel in 
sophomore and upper-division mathematics and science courses without the 
continued help of the project. 

This development of a classroom cohesiveness and feelings of companionship in the 
classroom was thought by the researchers to go along with the use of Cross/Angelo 
CATS which allow the instructor to collect student feelings about the class, share 
them with the class, and move together as a class to accomplish learning. 




ADDITIONAL RESEARCH ON CLASSROOM ASSESSMENT 
INFLUENCE ON DIVERSITY AND GENDER 

K. Patricia Cross, one of the originator's of the Cross/Angelo model, has talked about 
the student population that we teach in community colleges: 

Community colleges are the leading institutions in the hierarchy of higher 
education in serving more minorities, more part-time students, more 
adults, more blue- and pink-collar workers, more of the elderly, more 
single parents, and more displaced workers than any other segment of 
higher education. In short, every so-called "non-traditional" segment of 
society that was under-represented in higher education in 1950 is over- 
represented in community colleges today. 

The growth of minority enrollment in community colleges was recently discussed by 
the American Association of Higher Education in their report Minorities in Higher 
Education (Carter and Wilson, 1993). They state that there has been slight to 
moderate gains in college enrollment by minority students for the period 1990-1992 
with most of these gains occurring at the community college level. There was a 1 3.4 
percent gain at community colleges compared to a 5.9 percent increase at four-year 
institutions. 

To address the needs of diverse students. Cross suggests that classroom teachers 
become classroom researchers to systematically assess student learning, and, more 
specifically, to assess how students are responding to our efforts to teach them 
(1991). Cross further suggests that community colleges are good places to study 
diversity issues and needs as the community college structure welcomes a greater 
proportion of such a diverse population than do other segments of higher education 
(Cross,1991, AAHE, 1993). 

Hart (1992) addresses some of the questions raised by Cross when he looks at the 
diverse cultural and racial identities and needs of (multicultural) students and suggests 
that "students and the institution must make changes to create a mutually satisfactory 
environment" for learning. He proposes a seven step methodology to move toward 
an envi onmeric that serves more diverse needs. These steps are: 

1 . Select educational values. 

2. Translate values into specific goals. 

3. Design environments that contain mechanisms to reach stated goals. 

4. Make sure environments are fitted to diverse student needs. 

5. Measure student perceptions of the environment. 

6. Monitor student behavior resulting from their perceptions of the environment. 

7. Gather student feedback on the environment from student behavior and 
perceptions, then reassess and modify where necessary. 
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A study done by Obler, Arnold, Sigala, and Um')denstock (1931) reflects both the 
model described by Hart and the work of Uri Triesman. Four principles directed their 
work on increasing respect for diversity in classrooms at community colleges: 

1. Create a Pro-Diversity Curriculum defined as "increaseld) student awareness, 
skill, and applications of course content to a diverse, multicultural society. 

2. Treat students as capable but inexperienced adult learners. 

3. Promote cooperative learning. 

4. Use classroom assessment. 

Obler et.al. were concerned with both Pro-Diversity goals and Anti-Bias. Anti-Bias is 
defined as "challenging the prejudice stereotyping, and subt; ; or obvious biases in 
teaching methods or classroom resources." What this means in terms of teaching is 
that a conscious effort is made to eliminate negative references to groups and/or 
cultures and additionally, to help students identify such overt or subtle biases in 
material that they encounter. What these authors found was that both students and 
faculty were happy with subsequent changes made in curriculum. When students 
were assessed using Cross/Angelo Classroom Assessment Techniques, the feedback 
revealed positive student comments relevant to the new curriculum and teaching 
process employed. Students also gave feedback that they appreciated the 
cooperative learning aspects of the classes, as they felt that getting input from many 
diverse views was a very valuable part of their learning process. Faculty commented 
that they were impressed with the project and that students benefitted from attempts 
to include more diverse views in the curriculum. Obler also reports retention improved 
in classes where these principles were employed. 

The text Minorities on Campus: A Handbook for Enh ancing Diversity (Green, 1989) 
states that effective teaching that enhances learning for minority students includes 
"active involvement, frequent feedback, and understanding different ways of 
learning." The classroom assessment method meets these criteria. 

Theobald-Osborne (1991) studied adult learners and predictor variables for success. 
She found that "Faculty interaction was the greatest predictor of positive educational 
outcomes for adult students." The classroom assessment process of faculty asking 
for student feedback and then sharing with them the results is an interaction 
considered strongly positive by faculty and students both. 

A study recently released by the American Association of University Women called 
How Schools Shortchange Girls also echoes these sentiments. "Support from 
teachers can make a big difference. Studies report that girls rate teache. support as 
an important factor in decisions to pursue scientific and technological careers." 
Recommended actions for change to improve the success of women in higher 
education include the following statements: 
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School-improvement efforts must include a focus on the ongoing 
professional development of teachers ...Support and released time must 
be provided by school districts for teacher-initiated research on curricula 
and classroom variables that affect student learning. Gender equity 
should be the focus of this research and a criterion for awarding funds. 

The research conducted to examine the effect of the Cross/Angelo Classroom 
Assessment Model on student outcome may show that the model helps women 
students to succeed. 
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CHAPTER THREE 



METHODOLOGY 



RESEARCH DESIGN 



In this study, six student outcome variables were examined. This was accomplished 
by means of two designs, each of which was piloted, done on a small scale, and then 
incorporated into one large inclusive research design. In all cases, data was analyzed 
by grouping all eight colleges, and no individual students or instructors were identified. 

The first experimental model was a time series design or pre-experimental, one in 
which data was gathered on groups of students pre and post the instructor's training 
in the Cross/Angelo model. Classes were matched for same instructor, same subject, 
same course code, same time of day, but during different semesters. See Figure 1 
below: 



PRE AND POST RETENTION DATA STUDY 



AN INSTRUCTOR IS ENROLLED IN THE CROSS/ANGELO CLASSROOiVI 
ASSESSMENT TRAINING COURSE AT ONE OF THE EIGHT COMMUNITY 
COLt .EGES IN THE STUDY. 



THIS PERSON IS ATTENDING THE SESSIONS AND HAS A GOOD GRASP OF 
THE MODEL. S/HE DESIGNATES A PARTICULAR COURSE IN WHICH S/HE 
USES FEEDBACK TECHNIQUES. 



IN THIS DESIGNATED CLASS, S/HE USES THE TECHNIQUES CONSISTENTLY 
CONTINUOUSLY, AND APPROPRIATELY. 



RETENTION DATA IS THEN PULLED FOR THIS SAME COURSE TAUGHT BY 
THE SAME INSTRUCTOR IN THE SAME LOCATION AT APPROXIMATELY THE 
SAME TIME, ONE YEAR OR ONE SEMESTER PRIOR. 



AS INSTRUCTOR WHO IS IN TRAINING FINISHES THE CLASS, THE SAME 
RETENTION DATA IS PULLED FOR IMS COURSE IN WHICH THE INSTRUCTOR 
HAS USED THE FEEDBACK TECHNIQUES, APPROXIMATELY ONE YEAR AFTER 
THE FIRST SET. 



A COMPARISON IS MADE FOR RETENTION AND/OR GRADE STATISTICS PRE 
AND POST CLASSROOM ASSESSMENT TRAINING. 
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The second design used was quasi-experimental; one in which different groups were 
exposed to different treatment, but the groups were preexisting, and the differences 
in the groups were based on the existence of an independent variable. Tho 
independent variable here was student exposure to the use or non-use of the 
Cross/Angelo Classroom Assessment Techniques in their classes. Data was gathered 
on students in matched classes in which the instructor was either usir.a or withholding 
the Classroom Assessment Techniques. See Figure 2 below: 



MATCHED CLASS STUDY 
INSTRUCTORS HAVE BEEN TRAINED IN THE CLASSROOM 

ASSESSMENT MODEL 



ONE SECTION 
USES 

FEEDBACK TECHNIQUES 


ONE SECTION 
WITHHOLDS 
FEEDBACK TECHNIQUES 


SAME COURSE 
SAME INSTRUCTOR 
SAME BASIC TIME 
SAME SEMESTER 


SAME COURSE 
SAME INSTRUCTOR 
SAME BASIC TIME 
SAME SEMESTER 


CLASSROOM ENVIRONMENT 
INVENTORY USED TO SURVEY 
STUDENTS ON TENTH WEEK 


CLASSROOM EN' RONME^^IT 
INVENTORY USED TO SURVEY 
STUDENTS ON TENTH 
WEEK 


COMPLETE RETENTION DATA IS 

COLLECTED 

1ST CENSUS ENROLLMENT 
END OF TERM ENROLLMENT 
GPA 

GRADE DISTRIBUTION 
ETHNICITY AND GENDER 

INSTRUCTOR OPINION IS 
COLLECTED 


COMPLETE RETENTION DATA IS 

COLLECTED 

1ST CENSUS ENROLLMENT 
END OF TERM ENROLLMENT 
GPA 

GRADE DISTRIBUTION 
ETHNICITY AND GENDER 

INSTRUCTOR OPINION IS 
COLLECTED 
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MEASUREMENT AND INSTRUMENT SELECTION 

In the two research designs, a total of six different measurements of student outcome 
were used. Each measurement will be described. 

Retention studies were carried out by matching first census enrollment with end of 
term enrollment and obtaining the percentage of students remaining in the class. 

Grade point average was obtained by adding together the point values of all the 
grades distributed in a class (A = 4, B = 3, C = 2, D = 1, F=0, Cr = 2, No Cr = 0) and 
dividing by the number of students receiving a grade. 

Grade distribution was calculated by counting up how many of each grade was given 
in a class (number of A's, B's, C's, D's, F's, Cr, No Cr's and inc.'s) and reporting the 
total grade distribution. 

Retention by Ethnicity was calculated by identifying the ethnicity of each student 
enrolled at first census, counting how many of each ethnic group were enrolled 
(categories were Asian/Pacific Islanders, Alaskan/Native Americans, Black, White, 
Hispanic, Filipino, Other and No Answer) and comparing this to the percentages still 
enrolled at end of term. As this study had small numbers of some of the minorities 
listed here, a decision was made to compile all students categorized as Asian/Pacific 
Islanders, Alaskan/Native Americans, Black, Hispanic, Filipino, Other and No Answer 
into a category called New Majority and to report the data in this manner. 

Retention by Gender was calculated by identifying each enrolled student's gender at 
first census and comparing this to the percentages of men and women still enrolled 
at end of term. 

Classroom Environment was measured by using the CUCEI instrument (College and 
University Classroom Environment Inventory) developed by Treaquest and Fraser in 
1988 in Western Australia. This is a 49 question survey which asks students their 
opinions of the class envi^'onment. The instrument was found to be valid and reliable 
when used in technical schools in Australia and at universities in the United States. 
This was the CUCEI survey's first use in American community colleges. The CUCEI 
instrument has a possible score of 1-5 points for each of the 49 questions. The 
lowest score (1) is the most desirable classroom environment choice. There are seven 
subscales which are groups of seven questions each which are contained in the 
categories of classroom Personalization, Involvement, Cohesiveness, Satisfaction, 
Task, innovation, and Individualization. Treagust and Fraser (1986) defined these 
terms as follows: 

• Individualization is the extent to which students are allowed to make decisions 
and are treated differently according to ability, interest, or rate of working. 

• Innovation is the extent to which an instructor uses new and unusual teaching 
techniques, activities and assignments. 
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• Involvement is the extent to which students participate in class. 

• Personalization Is an Individual student's opportunity to interact with the 
instructor around personal welfare issues. 

• Satisfaction is the extent to which a student enjoys a class. 

• Student Cohesiveness is the extent to which students know, help and are 
friendly toward each other. 

• Task Orientation is the extent to which class activities are well organized and 
clear. 



During the year that this research study was being conducted, it was also the 
responsibility of the authors to supervise the Cross/Angelo training program at the 
eight colleges. Upon completion of the training, a survey form was distributed to all 
the participating instructors. Instructors were asked to anonymously answer the 
following questions: 



1 . Was the training a good use of your time? 

2. Were you able to meet new instructors on your campus and develop a sense 
of collegiality that might not have occurred otherwise? 

3. Do you think that asking students for feedback makes you a better teacher? 

4. How did your students respond to being asked for their feedback? 

5. What was your best experience with Classroom Assessment in your classes? 

6. Were there any negative experiences? 

7. Do you think that Classroom Assessment can help women and minority 
students succeed in classes that they might otherwise have dropped or failed? 

8. Do you think that this program should be continued at your school? 

9. Is there anything that you can do to support the continuation of this program? 

Additional Comments: 
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SELECTION OF SUBJECTS 



For the retention study pilot in Summer, 1992, three schools participated. The project 
leaders at these three schools randomly selected four instructors per school that had 
been trained in the Cross/Angelo Classroom Assessment Training during the previous 
school year. Random selection was done by putting the names of those who had 
completed the training in 1 991 -1 992 in a hat, and selecting four instructors by pulling 
four names out, one at a time, with all the other names returned to the hat on each 
draw. The class in which each instructor had used classroom assessment techniques 
during the 1991-1992 school year was identified. First Census Enrollment and End 
of Term grade sheet records were obtained for that class and for a matched class 
(same time, same course, one year or one semester earlier, prior to Cross/Angelo 
training.) Twelve instructors' retention data were examined in this pilot study. 

For the Fall, 1 992 retention studies, once again random selection was used to identify 
two instructors per college who were currently enrolled in Cross/Angelo training. Data 
was collected for these instructors for the identified class that they were currently 
teaching and using CATs in, and for a matched class one semester to one year prior. 
Twelve instructors representing six colleges participated in this study. 

For the Fall, 1992 matched class pilot studies in which an instructor was used a3 
his/bar own control, i.e., s/he used techniques in one class and withheld techniques 
in a matched class, the sample was taken from volunteers. Instructors to participate 
were found through personal contact or by notice in the college newsletter. It was 
not difficult to find volunteers who had two sections of the same course who had 
been trained in classroom assessment, but it was quite difficult to find such 
instructors who were willing to withhold the CATs from their second class. Even 
those who did agree to be subjects in the study had significant problems in this area, 
which will be discussed further in the pilot study report. Thirteen instructors from six 
colleges began the study, eleven completed it. 

In Fall of 1992, it was decided to add an additional control group of instructors who 
had never been trained in the Cross/Angelo method and who were not using CATs in 
their classrooms. Fifteen additional classes of eleven such instructors were surveyed 
at one college. An attempt was made to survey a variety of disciplines, and these 
fifteen classes consisted of Humanities, English Composition, Film Technology, 
Computer Science, Physics, Viticulture, Color and Design, Painting, Chemistry, 
Business and Psychology. 

In the final research design, in Spring, 1993, once again the sample consisted of 
volunteer instructors who were willing to teach one section of the same course using 
CATs and one section of the course withholding CATs. Initially two instructors from 
six schools and three instructors from two schools began the study, for a total of 18 
matched classes. Matched class data was used for fifteen of these instructors. 
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For the additional instructor survey study, the project directors at the eight colleges 
were given ten surveys to distribute to faculty who had been trained in the 
Cross/Angelo method during the 1992-1993 school year. 

In some cases, all eight schools are not represented in each study. Reasons that this 
occurred include: 

1 . Instructor moved away during study. 

2. Instructor had class canceled. 

3. Designated research director did not follow protocol instructions or missed data 
submission deadlines. 

4. Instructor desired to drop out of study due to ethical concerns of not wanting 
to withhold the CATs techniques. 

5. Instructor was unable to demonstrate consistent and reliable use of the CATs 
techniques in classroom. 
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CONCERN FOR THE RIGHTS OF HUMAN SUBJECTS 



Various measures were used to assure the protection of human subjects. Any records 
pulled were entered anonymously Into the computer. No names of students or 
instructors were included (see letter to admissions and records, page 63). Students 
answering the CUCEI inventory questions did so anonymously with no indication of 
their names or those of their instructors. Students were given the choice of 
participating or not participating in the data collection (see informed consent on page 
65). Records were kept locked in the home of the researcher and no one else had 
access to the data. Post Hoc data collected from Admissions and Records 
departments was done without instructor or student names attached (Addendum, 
page 64). All instructors whose classes were used in the study gave their consent 
to participate. The data was analyzed by school and by group and could not be 
separated to identify individual classes, instructors or students. In all cases, schools 
were designated as "School 1-8" and will be referred to as such in the reporting 
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CHAPTER FOUR 



PILOT STUDIES 



35 



PILOT TESTING 



Each of the eight schools had an assigned paid project director and paid research 
assistant. Meetings and correspondences took place regularly throughout the project 
year. Directions for the research protocol were given in person, in writing and on 
audio tape. In response to problems identified during the pilot testing, additional 
directions included a signed contract, a wall poster and more personal meetings with 
the participating instructors and research data collectors. 

RETENTION STUDIES, PRE AND POST ASSESSMENT TRAINING 

During the summer of 1992, retention data was collected at three colleges for a total 
of 1 2 instructors. The pilot was done ex post facto, by looking at previously compiled 
data from the admissions and records departments at three colleges. These records 
were used for a Pre and Post Training Study, in which Cross/Angelo trained 
instructors' end of term retention statistics were compared for the same class at the 
same time by the same instructor in the same location one year prior to training. In 
this pilot study, results indicated an increase from 80% end of term retention pre 
model training to 87% retention when CATs were applied. See Figure 3 on page 36. 



RELIABILITY OF RETENTION MEASURES 

Each school had an assigned "researcher" whose job it was to submit the necessary 
data to the authors. "Researchers" were given written instructions and each 
researcher was met with on two occasions to explain what was needed. This proved 
more difficult than anticipated. The original intention was to collect all retention and 
grade information directly from Admissions and Records Offices or Data Processing 
Offices that compile official student record data. The authors met with each college's 
Admissions and Records Dean, each of whom agreed to provide the necessary data. 
This proved possible, however, at only four of the eight schools. As was mentioned 
in the section on limitations of the study, this was a difficult problem at the other 
schools, whose management of information systems were not yet state-of-the-art. 
The authors therefore asked the assigned researchers to provide copies of 
participating instructors personal enrollment sheets that recorded class enrollment, 
official drops and grades. The retention data was mailed to the research director and 
then entered by the research director into the StatView program on a Mcintosh 
computer. It was analyzed with the SPSS (Statistical Package for Social Sciences) 
program. 




VALIDITY OF RETENTION MEASURES 



Retention and grades are two of the new Accountability Indicators that have been 
requested of colleges by the California Conrimunity College Chancellor's Office (Fetler, 
1993). The requirements report is divided into five sub-sets, those of Student 
Access, Student Success, Student Satisfaction, Staff Composition, and Fiscal 
Condition Under the area of Student Success, colleges are being asked to report how 
many students completed each course with a "C" grade or better. Colleges must look 
at the retention and the grade distributions in their classes. These appeared to be 
valid indicators to examine in relationship to a new teaching technique. 

RESPONSES TO THE RETENTION DATA PILOT 

This proved to be a reliable data collection method with statistical significance, so the 

witTnnt "^^^^^V this model of pre and post training analysis 
Without change in the design. 
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CLASSROOM ENVIRONMENT MATCHED CLASS PILOT STUDY 

A pilot matched class study was done during the Fall of 1992 to test the use of the 
CUCEI (College and University Classroom Environment Inventory) instrument. 
Instructors were asked to use Classroom Assessment Techniques frequently in one 
class and to completely withhold the CATs from the other class. Differences In 
classroom environment were to be measured by surveying classes in which the 
instructor was using Classroom Assessment Techniques in comparison with those 
classes offered in which the same instructor teaching the same course at the same 
time was not using any CATs. The CUCEI was conducted during the tenth week of 
those colleges on the semester schedule. This date was chosen as it fell after 
midterm exams and before final exams. Permission to use the instrument was 
obtained from the authors in Australia (Addendum, page 69). The instrument was 
retyped and a cover sheet added (Addendum, page 65). At each school, 
administering research faculty were provided with the necessary number of pre- 
addressed and coded envelopes (Feedback or Non-Feedback), with 100 copies of the 
CUCEI instrument and with 100 scantron forms. 

The survey was administered by the designated researcher at each college, and 
protocol for the administration of the survey was given to the researcher both verbally 
and in writing by the authors (Addendum, page 70). The researcher made 
appointments with each participating instructor for a day and time to come into the 
classroom. Each researcher administering the survey had the same set of directions 
to read to the students. A National Computer Systems Survey Sheet Number 1 6842 
was provided and Number 2 pencils were made available. The NCS forms were 
preceded to identify the school and to identify whether the class was a Feedback or 
Non-Feedback class. Students were told that the survey was anonymous, that neither 
they nor their instructor could be identified in any way, and asked not to fill in any of 
the identification blanks on the form. Students were also told that participation was 
voluntary and they could choose not to participate in the study with no reflection of 
non-participation on their grades. Instructors were asked to leave the class'room and 
20 minutes was allotted to complete the survey. All participating instructors were 
given a written thank you and assurance that all responses would be confidential and 
that their names would not be included on the data collected (Addendum, page 71 ). 
It was interesting to note that most instructors were disappointed that their own 
results would not be identified and shared with them. 

The CUCEI's were administered to a total of 624 students in eleven matched classes 
The NCS Forms were collected, and put into the coded and pre-addressed envelopes 
They were then mailed by the schools to the project director. Upon arrival the NCS 
forms were scored on a Hewlitt Packard Sentry 3000 Scanning machine 'using the 
Topscore program. The data was then entered into an SPSS program for analysis 
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VALIDITY AND RELIABILITY OF CUCEI INSTRUMENT 

The Colleqe and University Classroom Environme--.* Inventory was developed by Dr^ 
B eraser and Dr. David Treagust of the Curtin Institute of Technology Western 
AustraTa This tool has been used throughout Australia and also in severa Amer.can 
CiuHiPrtwXra 1979) Treagust and Fraser identified elements of desirable 
das oom eSnmem reviewing all prior published instruments. They estabhshed 
content validity based on categories of human environment well suited to the 
classroom as dentified by Moos (1974). Moos included elements of environmen 
Such as the nature and intensity of personal relationships, dimensions of persor^al 
growth and self-enhancement, and the extent to which the 

c ea7 in expectation, maintains control and is responsive to change. Numbers of 
?^cu ty and stude^^^^^ in Australia were interviewed to assess the validity of the too . 
Each rev^ion passed through evaluation by experts in the fields of questionnaire 
construcrn and college level teaching. The final product was a 49 qu t - 
with seven subscales. These scales were measures of desirable items in a college 
dassroom The scales measure the constructs of personalization involvement, 
studemcohesiveness, satisfaction, task or-^ 

Reliability was established in Australia by using Cronbach's alpha coefficient tests on 
each of t^e 49 items when expressed to four different groups (student actual 
response student desired response, instructor actual response, instructor desired 
response) ^34 classes. Internal consistency of the elements in the group means 
ranged from .78 to .96. 

Since development, this questionnaire has been accepted as a reliable and valid 
measure of desirable classroom traits which enhance learning. 

RESULTS OF CUCEI MATCHED CLASS PILOT 

For this pilot exam, the CUCEI did not reveal any statistical difference, or in fact any 
differences at all between the two classes from the same instructor, one class which 
was receiving CATs and one dass which was not. See Figure 4 on page 42. 
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RESPONSES TO CUCEI PILOT RESULTS 

There were three responses to this CUCEI pilot data. 

1. At first, the instrument was thought to be an invalid measure to ascertain 
differences in community college classrooms, and consideration was given to 
discarding it. The classroom environment literature was again reviewed and the 
authors of the instrument in Australia were contacted. They affirmed that if 
there were differences in the classroom environments, that this instrument 
should be sensitive enough to pick them up. 

2. The researchers then questioned whether the participating instructors had 
followed the research protocol precisely. Upon questioning the instructors, it 
was found that nearly all participants had corrupted the protocol in some way. 
Some instructors admitted to using the responses given in the feedback class 
to make changes in the non-feedback class. Other instructors stated that they 
used feedback in both classes out of ethical concerns, and indeed one school 
had to be eliminated from this part of the study because of this. A few 
instructors stated that they used only a minimal amount of feedbacktechniques 
in the feedback class. It was found that there were wide ranges in the number 
of times that feedback was used between the different participants. The 
researchers realized that a much tighter protocol with much clearer directions 
would have to be developed in order to ensure the integrity of the study. 

3. A third response to the results of the CUCEI pilot was the supposition that 
perhaps when instructors are trained in the Cross/Angelo method and accept 
the premise of a shared teaching/learning partnership with students, that 
conceivably perhaps they undergo an intrinsic change in behavior that is 
communicated io students whether actual assessment techniques are used or 
not. This could explain the absolute lack of difference in responses between 
trained instructor's matched classes. 

Two actions were taken in response to the lack of difference in the CUCEI survey 
results. 

1. An administration of the CUCEI survey was done in 15 additional classes in 
which the instructors had never been exposed to the Cross/Angelo training. 
This group was named the Additional Control Group. The data was then 
analyzed comparing the trained instructors using the CATs in their classrooms 
with the never trained instructors who did not use CATs in their classrooms. 
Out of this comparison, one statistically significant difference was identified 
between those two groups. The subscale measure of classroom cohesiveness, 
a factor known to contribute to retention (Triesman, 1985), was found to be 
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different between the classes using and not using the model. (See Figure 6.) 
Therefore the researchers decided to go ahead with the instrument, initiating new and 
very explicitly defined protocol instructions. 

2. To tighten the protocol, much more stringent standards were developed. Each 
faculty member participating in the Spring, 1993 study was given a contract 
to sign, a wall poster (Addendum, pages 72-73) delineating each week's 
activities (use of a minimum of one feedback technique every two weeks in the 
feedback class and absolutely no feedback techniques in the other), and a 
personal interview by the researchers. Two mandatory regional group meetings 
with the data collectors were held, with distinct instructions delivered. With 
these newly delineated responsibilities, the participants could be expected to 
carry out the research design. 
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ADDITIONAL RESULTS OF MATCHED CLASS STUDY, FALL, 1992 



As mentioned previously, between the matched classes of instructors trained in the 
method who used the method in one class but withheld it in the other, no differences 
were found on the classroom environment scales. In both cases, the students 
indicated the same satisfaction with this instructor's classroom environment. 
Instructors who participated in the matched class study were also asked to send the 
researchers their two final grade sheets and to fill out a questionnaire asking about 
their perceptions of the two classes (Addendum, page 74). Although some 
instructors indicated that they could feel a difference and some indicated they could 
not feel a difference between the classes, an important secondary finding occurred. 
When grade sheets were examined, a difference was found in the grade distributions. 
In non-feedback classes, 20% of the students received D or F grades. In feedback 
classes, instructors gave many fewer D's and F's (only 07%) and many more A, B and 
C grades (93%). A relationship appears to exist between the use of anonymous and 
continuous feedback and positive student grades. See Figure 7. 

An instructor opinion survey was sent to the eleven instructors participating in the 
matched class study. Seven instructors (64%) returned the survey. Instructors 
stated that students in the feedback classes more often came to class on time (25% 
more often), attended regularly (60% more often), came with their textbooks (33% 
more often), came with their homework done (50% more often), asked questions 
about the classroom material (100% more often), and used office hours more often 
(50%) than when compared to non-feedback classes. Non-feedback classes were 
identified as coming to class late more often, and remaining quieter in class when the 
instructor was talking. There were no differences in how often students came with 
paper to write on, brought pencils to class, or remained quiet while another student 
was talking. This survey indicated that in classes where CATs were being used, more 
positive student behaviors were perceived by the instructors. 
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Chapter Five 



Research Studies 



SECOND RETENTION STUDY 



After completion of the pilot testing, the authors then repeated the collection of pre 
and post retention data in a larger study during Fall of 1992. In this second study, 
six colleges participated and 12 classes were used for data coilection. Instructors 
were randomly selected from among those attending the classroom assessment 
training at the eight schools, who had taught the same class that they were using the 
CATS in, one year prior. Again retention data was compared between the matched 
classes pre and post training. In this study, the variable of class grade point average 
was added. 

In the Fall, 1992 study, consistent with previous retention studies, there was a slight 
increase in retention, from 78% pre-Classroom Assessment Training to 81% post 
training. When the data was looked at for each school separately, four of the six 
schools had increases in retention post training. When retention was analyzed by 
gender, males had a significant increase and females a slight increase in retention post 
training. Retention was also examined by ethnicity, and both new majority and white 
students increased in retention post training. These results are displayed on Figures 
6 and 7. Similar to the study at College of Marin (Stetson, 1983), there was no 
increase in grade point average noted, as displayed on Figure 6a. 
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FINAL RESEARCH DESIGN 



The final matched class research design combined elements from the pilot and initial 
small scale studies. It examined retention overall, retention by ethnicity and gender, 
and examined grade distribution and grade point averages. The study included the use 
of the CUCEI inventory and a survey to instructors asking their opinions of the use of 
the Cross/Angelo method. See Figure 8 below: 

Figure 8 
SPRING 93 FINAL DESIGN 
MATCHED CLASS STUDY 
ALL INSTRUCTORS TRAINED IN CLASSROOM ASSESSMENT TECHNIQUES 



ONE SECTION 
USES 

FEEDBACK TECHNIQUES 


ONE SECTION 
WITHHOLDS 

VVI 1 1 II IV^L_Liyw 

FEEDBACK TECHNIQUES 


SAME COURSE 
SAME INSTRUCTOR 
SAME BASIC TIME 
SAME SEMESTER 


SAME COURSE 
SAME INSTRUCTOR 
SAME BASIC TIME 
SAME SEMESTER 


CUCEI DATA COLLECTED WEEK 10 


CUCEI DATA COLLECTED WEEK 10 


COURSE IS COMPLETED 

COMPLETE RETENTION DATA IS 
COLLECTED 

1ST CENSUS, END OF TERM, 
GPA, GRADE DISTRIBUTION, 
ETHNICITY, GENDER 


COURSE IS COMPLETED 

COMPLETE RETENTION DATA IS 
COLLECTED 

1ST CENSUS, END OF TERM, 
GPA, GRADE DISTRIBUTION, 
ETHNICITY, GENDER 



Additional Design 



INSTRUCTOR SURVEYS DISTRIBUTED TO ALL FACULTY TRAINED IN 
CROSS/ANGELO METHOD DURING GRANT YEAR 1992-1993 
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For the final study, as described earlier, a very strict protocol was developed to show 
instructors exactly what to do. Every participant signed a written contract 
(Addendum, pages 75-76). 

The designated school researcher collected first census enrollment sheets from the 
participants. On these sheets, ethnicity and gender were identified. Percentages of 
men, women, Asian/Pacific Islanders, Alaskan/Native Americans, Black, White, 
Hispanic, Filipino, Other and No Answer were calculated. The data was mailed to the 
research director. The researcher entered the data on a Mcintosh Computer and 
analyzed it using the StatView program. Because numbers of individual minorities 
were small, the researcher combined all ethnic groups other than White into an overall 
group called New Majority students. 

On the tenth week of the semester, the CUCEI surveys were administered to students 
in all classrooms. In an attempt to answer a question posed by the California 
Community College Chancellor's Office, i.e., does classroom assessment help women 
and minorities succeed in math, science and technology classes, three- anonymous 
demographic questions were added to the survey. Students were asked their gender, 
ethnicity, and whether the class they were in was a math, science, vocational or 
technology class. The data was then analyzed not only by feedback group and non- 
feedback group, but by specific gender and ethnic subsets within the groups, and by 
type of class. When analyzing the CUCEi data, the feedback class data was also 
compared with the additional control group of untrained instructors developed as part 
of the pilot exam. 

After the final date to drop, approximately the twelfth week, census sheets were once 
again collected by the researchers at each school. Calculations were made for 
retention overall, retention by gender, and retention by ethnicity. 

At the semester's end, grade sheets were collected and mailed to the research 
director. Class grade point average and grade distributions were calculated. 

Surveys were sent to all instructors who were trained in the Cross/Angelo method 
during 1992-1993 to ask their opinions of the effect of the method in their 
classrooms. 
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TESTS USED FOR DATA ANALYSIS 



To calculate retention comparisons, data was analyzed using StatView 512+ on a 
Macintosh computer. Means, standard deviations, percentages, column totals and 
effect sizes were computed. 

Effect size was calculated using the difference between the means fur each treatment 
group, and the standard deviation for the no feedback group. 

To calculate grc je point average, StatView 512+ on the Macintosh computer was 
used to calculate means, standard deviations and percentages. 

To analyze grade distribution, the calculator on the Macintosh computer was used to 
do addition, multiplication and percentages for each grade subsec in each treatment 
group. 

To analyze the CUCEI data, unmatched t-tests were done for the feedback and non- 
feedback group for the seven subscales. Unmatched t-tests were also run for subsets 
of the population which included males, females, whites, new majority, and students 
enrolled in vocational education classes. Two-Way Analysis of Variance was done for 
each of the groups named above. Additionally, effect sizes were computed for t-test 
results. The CUCEI results wer. analyzed using SPSS (Statistical Package for Social 
Sciences) on an I.B.M. computer. To develop the category of new majority students, 
individual subsets of ethnic data were collapsed into a larger category as each sub-set 
was toe small to analyze separately. 

To tabulate instructor opinion of the differences in their matched classes, a handheld 
calculator was used to calculate means and percentages. 

To tabulate all trained instructor opinion, surveys were read and all responses were 
tabulated by hand. 
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RESULTS OF RESEARCH STUDY 



During the Spring of 1993, tlie study design was inclusive of all previous tests. Six 
schools participated in this study. The overall end of term retention was the weakest 
for the three retention studies done (Pilot Fall, 1 992; Spring, 1 993). Overall retention 
went up only one percent, and went up significantly only at one school. Retention by 
gender in this study had very strong results, with female students in classes using the 
Cross/Angelo method reflecting an additional nine percent increase over the norm. 
Retention did not change for men, new majority or white students. 

Grade point average was higher overall in classes using the Cross/Angelo method, b jt 
not significantly higher. 

Grade distribution showed differences, but not as strongly as in the Fall, 1 992 study. 
In this case, overall distribution was about the same, but feedback classes showed 
5% more "A" grades. See Figure 9. This is consistent with a larger number of "A" 
grades in the feedback classes occurring in the Fall, 1992 study. 

The CUCEI findings for the Spring, 1993 study were consistent with the Fall study. 
Once again, there was very little difference when Cross/Angelo instructors were 
measured against themselves. There were, however, major differences when the 
trained instructors were matched to the additional control group. There were 
statistical significances found in the compiled CUCEI scores and in six of the seven 
of the sub-scales. These differences are especially important as the protocol for the 
trained instructors was clearly followed this semester. These findings strongly 
suggest that when CATs are consistently and reliably applied, there are major 
differences in how students feel about the classroom environment, significant at the 
p = .000 level. 

The CUCEI results were also analyzed with two-way analysis of variance. The 
findings here indicated a very strong connection atween the use of CATs and a 
positive learning experience for women and new majority students. When data was 
analyzed by treatment (CATs) with ethnicity, there were main effects for ethnicity cn 
the subscales Task (F = 8.726, p = .003) and Personalization (F = 9.796, p = .002). 
There was an interaction effect for treatment (CATs) with ethnicity on the subscale 
Cohesiveness (F = 4.492, p = .034). Interaction effects at these levels suggest that 
we can safely assume that new majority students felt better in these CATs classes 
and that this fact cannot be attributed to chance alone. 

When data was analyzed by treatment (CATs) with women students, mere were 
several areas of main effect. Women students felt more involved in the class 
(F = 3.267, p = .039) and more satisfied (F = 4.474, p = .035). 
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When the data was analyzed for use of CATs with new majority women students, 
very significant findings arose. These students felt that their learning was 
personalized (F = 1 5.852, p = .000), and overall they rated the classroom environment 
much higher than in classes they took where CATs were not used {Compiled Score 
F = 4.648, p = .032). 
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Figure 9 
RESULTS OF FINAL STUDY 
SPRING 1993 
Matched classes Different sections 
Sarr.e course, same instructor, same time, same place 
All CATS trained instructors 



Group 
Census 


Non-Use of Techniques 
(N = 488 students, 15 
classes) 


Use of Techniques 
(N = 492 students, 15 
classes) 


End of Term Retention 


78% 


79% 


End of Term Retention Women 


76% 


85% 


End of Term Retention Men 


80% 


76% 


End of Term Retention New 
Majority 


79% 


78% 


End of Term Retention White 


80% 


79% 


Mean Grade Point Average 


2.49 (.493) 


2.58 (.4871 


A, B, C Grades 


82% 


83% 


%D, F Grades 


18% 


17% 


Retention by School 

School 1 


.879 


.858 


School 3 


.765 


.708 


School 4 


.581 


.835 


School 5 


.866 


.865 


School 6 


.738 


.726 


1 School 7 


.674 


.584 
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ADDITIONAL INSTRUCTOR SURVEY 



A survey form was distributed to ail eighty instructors at the eight colleges who were 
trained in the Cross/Angelo model during the 1992-1993 school year. Forty-six 
questionnaires (56%) were returned. Instructors anonymously answered the following 
questions: 

1. Was the training a good use of your time? 

All 46 instructors felt that the Cross/Angelo training was a good use of their 
time. 

2. Were you able to meet new instructors on your campus and develop a sense 
of collegiality that might not have occurred otherwise? 

Instructors stated that they were able to meet new people or spend time with 
people that they already knew for the purpose of talking about teaching. Many 
instructors indicated that this was the best part of the training. 

3. Do you think that asking students for feedback makes you a better teacher? 

Most instructors felt that asking for feedback made one a much better teacben 
One instructor wrote "It is not so much my asking for feedback as r^y 
responding to the feedback that makes me better," 

4. How did your students respond to being asked for their feedback? 

Ninety percent of the faculty stated that students responded positively. Ten 
percent stated that students occasionally felt that this was wasting valuable 
class time. 

5. What was your best experience with Classroom Assessment in your classes? 

Instructors cited many interesting positive things that had happened to them 
as a result of asking for and responding to feedback. One person wrote "My 
best experience was watching the smiles on student faces when I reported 
back to them the results of their feedback." 

6. Were there any negative experiences? 

Forty-five out of forty-six instructors (98%) indicated that there were absolutely 
no negative experiences with the use of Classroom Assessment Techniques. 
Only one (2%) stated that the students felt time in class was being wasted. 
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7. Do you think that Classroom Assessment can help women and minority 
students succeed in classes that they might otherwise have dropped or failed? 

Instructors indicated that allowing women and minority students the chance to 
give anonymous feedback which would allow them to have points clarified may 
have had an influence on minority retention. Instructors felt that without the 
anonymous cards these groups might have been afraid to speak up and get 
help. 

8. Do you think that this program should be continued at your school? 

All those responding said yes. One person wrote that s/he would like to know 
how much the program cost. 

9. Is there anything that you can do to support the continuation of this program? 

All respondents stated that they would "talk this program up." Some 
suggested that this be an ongoing flex day activity. 

Additional Comments: 

The additional comments made by faculty were compliments about the 
individual project leaders at their school. Respondents felt that the leaders had 
been committed, professional and inspirational teachers of the model, and had 
conducted sessions that were most instrumental to professional growth. 



64 

56 



DISCUSSION 



The California Community College Chancellor's Office, the Federal Government, and 
many individual college districts have funded dissemination of and training in the 
Cross/Angelo model of Classroom Assessment. Qualitative and anecdotal reports 
about the Cross/Angelo CATs method has been consistently positive in nature. 

During the year prior to and the year of the research project, the authors of this report 
traveled extensively, disseminating information about the model and conducting 
training sessions, in every instance, audiences responded to what was presented 
positively, as was evidenced by their questions, requests for training information and 
for the results of this study when they were available. As the model acceptance 
grew, researchers attempted to link the use of CATs in the classroom to positive 
student outcomes. 

In the Spring of 1992, the California Community College Chancellor's Office 
expressed interest in exploring whether quantitative data would support the positive 
results of the qualitative and anecdotal reports. This study was funded by the 
Chancellor's Office to investigate the possibility. The Chancellor's Office additionally 
questioned whether the use of CATs could improve women and minority retention in 
math, science and technology classes. 

The results of this study do support the qualitative and anecdotal results as previously 
collected. However, the quantitative results are not as dramatic as qualitative or 
anecdotal data. What was found, using as strict a protocol as working with 70 
teachers at eight different sites can be, was as follows: 

1. Does the use of Classroom Assessment Techniques have an effect on 
retention? 

Consistent with previous studies, the use of Classroom Assessment Techniques 
to solicit student feedback appears to have an effect on retention, which 
ranged in our study from between 1 to 8 percent. 

2. Does the use of Classroom Assessment Techniques have an effect on final 
grades and grade distribution? 

There appears to be a correlation with the use of the techniques and more 
positive grade distribution, especially in the number of "A" grades given when 
Classroom Assessment Techniques are used. Grade point averages have been 
about the same. 
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3. Does the use of Classroom Assessment Techniques have an effect on course 
completion by gender? 



Female students seemed to persist as evidenced by nine percent higher 
retention in classes where CATs were being used. Female students also had 
significant differences than males on their opinions of the classroom 
environment when CATs were used. This appeared on the CUCEl inventory in 
the overall compilation and in the subsets of personal, involvement, satisfaction 
and individualization. Female new majority students in math, science and 
technology classes had a greater feeling of satisfaction and involvement than 
any other group. 

4. Does the use of Classroom Assessment Techniques have an effect on course 
completion by ethnicity? 

Although there did not appear to be a difference in the retention rates between 
new majority students and white students when examined by treatment, there 
was a clear difference in how new majority students felt about the classroom 
environment in classes using the methods. Positive statistical significance 
arose in areas of task understanding, cohesiveness, and personalization for new 
majority students. 

5. Does the use of Classroom Assessment Techniques influence the classroom 
environment? 

Students like it. There was increased student self report of class involvement, 
cohesiveness, personalization, satisfaction, task understanding, and instructor 
innovation in classes were the instructor was trained in and using the CATs as 
compared to classes where instructors were not trained in the method and 
CATs weru not used. 

Teachers consistently like it. Anonymous feedback collected from 46 
Cross/Angelo trained instructors reported 98% satisfaction with the use of the 
model in their classes and 2% of negative experiences with the model. As 
reported earlier, instructors liked using the method so well that many refused 
to withhold the CATs techniques from students. 

The null hypothesis, that there would be no differences in measurements of student 
outcome (retention, grade point average and grade distribution, course completion by 
gender and ethnicity and classroom environment) in courses where Classroom 
Assessment Techniques were used when compared to matched classes in which 
Classroom Assessment Techniques were not used, can be rejected in the areas of 
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overall retention, grade distribution, retention by gender, and classroom environment 
Retention differences by ethnicity did not occur, but strong differences in opinion of 
classroom environment were present. The null hypothesis cannot be rejected forihe 
area of grade point average. 

The researchers also felt that something appears to happen to instructors when they 
are Cross/Angelo trained. There was a clear difference in classroom environment 
between those who have volunteered for Cross/Angelo training and were using CATs 
in the classroom and those who did not volunteer for the training and were not using 
CATs. Whether the difference is that the very instructors who volunteer to learn 
something new to make them better teachers are different from those who do not 
volunteer, or the difference is that once instructors are open to the partnership with 
students they are intrinsically changed, we are not certain. We do know that when 
we measured the groups of Cross/Angelo trained instructors against themselves in 
both Fall of 1992 and Spring of 1993, there were no differences. Whether these 
groups were using 3X5 cards to collect anonymous feedback or not, students had 
Identical responses to them. This is an area that is commands further exploration. 



59 
67 



RECOMMENDATIONS 



1. There should be continued dissemination of this model to faculty in higher 
education. This should be particularly targeted to areas in which women 
students have traditionally not succeeded as it appears that the model has a 
positive effect on female retention. The model also seemed to benefit new 
majority students who felt a sense of personal ability to interact with the 
instructor and cohesiveness towards the group. We know from the work of 
Triesman that these findings are important. 

2. Faculty should continue to be trained by Cross and Angelo at the UC, Berkeley 
Summer Sessions for as long as they are offered, or at any other forum in 
which the authors of the model are doing the training. Additional 
recommended trainers are listed in the Catlin and Kalina manual (1993) found 
in the reference section. 

3. Instructional improvement projects which bring groups of instructors together 
for the purpose of talking about teaching should continue. Many faculty 
reported that this process resulted in a recommitment to teaching in ways that 
were very positive. 

4. The use of the Australian College and University Classroom Environment 
Inventory , the CUCEI, should be continued. This was the first use of the 
instrument in a California Community College setting, and it did appear to be 
quite useful in defining classroom environment. One suggestion might be to 
repeat the matched class studies and to add administration of the instrument 
to the faculty teaching the classes as well. Teacher opinion of the classroom 
environment was a recommended use by Treagust and Fraser, who felt that the 
closer the student's desired environment was to the teacher's desired 
environment, the more positive learning would take place. 

5. Although the purview of this study was not on the success of women in math 
and science, the finding suggests that CATs will help. Commentary is provided 
in the Addendum by Professor of Social Sciences, Lauren Coodley. 

6. As Angelo noted in his June, 1993, remarks, and as the authors noted in this 
report, something appears to happen to faculty when involved with the 
Cross/Angelo method. The literature suggests that "teacher effect" is the 
major influence on student success. What qualities constitute a "good teacher" 
and how faculty can learn these qualities is an area calling for further 
exploration, 
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Conducting a multi-site consortium research project requires clear directions 
and explicit protocols. Release time for participants rather than stipends, 
campus researcher involvement, larger than anticipated travel budgets to make 
frequent face-to-face visits, compatible computer systems, signed contracts for 
work to be done and payment tied directly to performance are some areasthat 
were brought to light. Positive findings were that multi-site work allows for a 
larger and more diverse "N" and group writing allows for expansion of ideas 
beyond what one or two people could create. The Infonet communication 
system functions extremely well for this type of group effort. These findings 
will be included in a manuscript in progress (Catlin, 1 993) and the authors hope 
that what they have learned during this year will benefit others planning to 
undertake such a project. 
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ADDENDUM 
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■Classroom 4sse,ssni^n( ?rm?t 

Napa Valley College 
2277 Napa-Vallejo Highway 
Napa, CA 94558 
(707) 253-3135 




September 9, 1992 

To: Admissions and Records Officer 

From: Michelle Kalina, Research Director 
Anita Catlin, Project Director 
LARC Classroom Assessment Grant 
FII-#92-0016 

Re: Retention Data 

the information listed below. P=^"cipat.ng ,n. We will need your help in providing 

^ns^SrLmeT''"'" '"""'"'^ ^' P-"^^ f°"--S: 
Course code: 
Section Number: 
Semester Taught: 

We will then need: 
First Census Enrollment 
Gender Distribution at First Census 
Ethnic Distribution at First census 

End of Term Enrollment 
End of Term Grade Mean 
Gender Distribution at End of Term 
Ethnic Distribution at End of Term 

All information will be held confidential. Instructor and student names will be removed from 
We thank you very much for your support of this statewide grant activity. 
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Napa Valley College 



Napa, California, 94558 
(707)253-3100 



OFFICE OF INSTRUCTION AND STUDENT SERVIc'eS 



TO: 



Admissions and Records Offices 

Participating Schools in Classroom Assessment Research Grant 
Napa Valley College, Grant Project Director 



FROM: 



DATE: 



J'jiy 27, 1992 



RE: 



Funds for Instructional Improvement-Chancellor's Office 



As a requirement of the research grant received from the Chancellor's Office, FII-92-001 8 
we will be collecting retention data for classes in which the instructors are using ihe 
Classroom Assessment process. Over the next school year, we will be asking you for 
limited data regarding student success. This semester, we will need first census lists and 
final grade reports for designated classes. 

Protection of Human Subjects will be assured, as we will not be using any names of 
instructors or students for our data collection, and the data from your college will be 
combined with the other eight colleges. We will be looking at the relationship if any 
between Classroom Assessment techniques, grades and retention. We may later do 
some statistics regarding ethnicity of students, but again, no names will be used. 

We greatly appreciate your cooperation in this project. The ultimate goal is to improve 




72 



ERIC 



64 



Classroom Assessment Project 



Dear Student, 

This research survey is a voluntary study of classroom environments 
being conducted throughout the State of California. Neither you nor your 
instructor will be able to be identified in any way. Your instructor will not 
see the results of this survey. Your status in class and grades will not be 
affected by either your participation or non-participation in the study. 
You have the right to refuse to participate. We thank you very much if 
you have chosen to participate. Your opinions will help us in the 
development of teacher training programs. 

Please feel free to ask the person administering the survey if you have 
any questions. 

Thank you! 
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COLLEGE AND UNIVERSITY 
CLASSROOM ENVIRONMENT INVENTORY (CUCEI) 



Developed by Dr. David F. Treagust and Dr. Barry J. Fraser 
Western Australia Institution of Technology 



Directions 



The purpose of this questionnaire is to find out your opinions about the class you 
are attending right now . 

This form of the questionnaire assesses your opinion about what this class is 
actually like. Indicate your opinion about each statement by darkening in the 
correct bubble: 



A - if you STRONGLY AGREE that it describes what this ciass is actually like. 

B - if you AGREE that it describes what this class is actually like. 

C - if yo'j DISAGREE that it describes what this class is actually like. 

D - if you STRONGLY DISAGREE that it describes what this class is actually like. 



• All responses should be on the attached ScanTron. 

• Do not put your name on the ScanTron. 

• Use a number "2" pencil. 

• If you must erase, please erase completely. 
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1 . The instructor considers students' feelings. 

2. The instructor talks rather than listens. 

3. The class is made up of individuals who don't know each other well. 

4. The students look forward to coming to classes. 

5. Students know exactly what has to be done in our class. 

6. New ideas are seldom tried out in this class. 

7. All students in the class are expected to do the same work, in the same way 
and in the same time. 

8. The instructor talks individually with students. 

9. Students put effort into what they do in classes. 

10. Each student knows the other members of the class by their first names. 

1 1 . Students are dissatisfied with what is done in the class. 

12. Getting a certain amount of work done is important in this class. 

13. New different ways of teaching are seldom used in this class. 

14. Students are generally allowed to work at their own pace. 

15. The instructor goes out of his/her way to help students. 

16. Students "clockwatch" in class. 

17. Friendships are made among students in this class. 

18. After the class, the students have a sense of satisfaction. 

19. The group often gets sidetracked instead of sticking to the point. 

20. The instructor thinks up innovative activities for students to do. 

21 . Students have a say in how class time is spent. 

22. The instructor helps each student who is having trouble with the work, 

23. Students in this class pay attention to what others are saying. 

24. Students don't have much chance to get to know each other in this c!ass. 

25. Classes are a waste of time. 

26. This is a disorganized class. 

27. Teaching approaches this class are characterized by innovation and 
variety. 

28. Students are allowed to choose activities and how they will work. 

29. The instructor seldom moves around the classroom to talk with students. 

30. Students seldom present their work to the class. 

31 . It takes a long time to get to know everybody by his/her first name in this 
class. 

32. Classes are boring. 

33. Class assignments are clear so everyone knows what to do. 

34. The seating in this class is arranged In the same way each week. 

35. Teaching approaches allow students to proceed at their own pace. 
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36. The instructor isn't interested in students' problems. 

37. There are opportunities for students to express opinions :n this class. 

38. Students in this class get to know each other well. 

39. Students enjoy going to this class. 

40. This class seldom starts on time. 

41 . The Instructor often thinks of unusual class activities. 

42. There is little opportunity for a student to pursue his/her particular interest in 
this class. 

43. The instructor is unfriendly and inconsiderate towards students. 

44. The instructor dominates class discussions. 

45. Students in this class aren't very interested in getting to know other 
students. 

46. Classes are interesting. 

47. Activities in this class are clearly and carefully planed. 

48. Students seem to do the same type of activities every class. 

49. It is the instructor who decides what will be done in our class. 

Questions about you 

50. Gender 

a. male 

b. female 

51. Ethnicity 

a. Alaskan/Native American 

b. Asian/Pacific Islander 

c. Black 

d. White 

e. Hispanic 

f. Filipino 

g. Other 

52. Is this class a math, science, technology, nursing or vocational training 
course? 

a. yes 

b. no 



Thank you very much for your assistance. If you desire further information, please 
contact either Dr. Michelle Kalina at Sierra College at 916-624-3333, ext. 2274, or 
Anita Catlin at Napa Valley College, 707-253-3135. 
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Classroom Assessment Project 

Napa Valley College 
2277 Napa-Vallejo Highway 
Napa, CA 94558 
(707) 253-3135 
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PROTOCOL FOR ADMINISTERING THE CUCEI INSTRUMENT-SPRING 93 

STATEWIDE Fll GRANT#92-0016 

1. The researcher will arrange an appropriate time to collect the CUCEI data during the 
tenth week of the semester with the two instructors involved in the matched class 
pilot study. 

2. When going in to administer the survey, each class and each instructor will have a 
specified envelope. It is imperative for the integrity of the research that these 
envelopes be administered properly. A further check can be on the third digit code 
number, 0 = No Feedback, 1 = Feedback techniques used. 

3. The researcher will bring the questionnaires, the scantrons, pencils and envelopes 
to the classroom. The researcher will distribute and collect all items upon 
completion. Please show students the scantron. (There may be students in the 
class who have never used a scantron.) Explain where the numbers 1-52 start and 
stop. Tell them that nothing else should be filled in by them except these 52 
answers. Remind them about erasing well. 

4. The researcher will read the disclaimer and directions printed on the front page of 
the CUCEI document and field all questions. The researcher should ask students to 
look over the instrument and ask for any word definitions that they do not 
understand. Words may be defined fT the students. 

5. There are now three additional questions on the form. Please help determine ■ 
whethe, this class is a math, science, technology, nursing or vocational training i 
class. We are defining vocational as any course which directly prepares a student 
tor a job skill, such as drafting, bookkeeping, art graphics, etc. i 

6. If there are students who do not wish to participate, they may remain quietly in \ 
Class or be excused by the instructor, per instructor preference. The instructor 
need not be present while the survey is being administered. 

7. The researcher may tell students about this instrument, develooed m Western 
Australia by Treagust and Fraser. We found the instrument by means of a library 
search and have obtained the developers' permission to use the instrument in our 
study, to measure as they did, students' opinions of their classes 
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Classroom Assessment Proj erl 
Napa Valley College 
2277 Napa-Vallejo Highway 
Napa, CA 94558 
(707) 253-3135 

September 4, 1992 




TO: All Instructors participating in the CAP research project 
FROM: Michelle Kalina and Anita Catlin 

Thank you for allowing your students to participate in the CAP research project. Your 
volunteering to help with this research is much appreciated by us and the Chancellor's 
office (our funding source). 

This is to assure you that all data collected will nsd be identifiable by either instructor 
names of students or school and will only be analyzed as a group of 8 community 
colleges. Individual data will not be shared with administrators and only group means 
and standard deviations will be reported. 

Due to this, we won't be able to share your own classroom results with you as we are 
not analyzing them by class or by school. However, you can run a similar survey 
yourself to get your own data at another time. 

If you have any question or concerns, please contact Michelle Kalina at 916-624-3333 
ext. 2274. 
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A QUICK CHECKLIST 
FOR PARTICIPATION IN THE 
CLASSROOM ASSESSMENT STUDY 

□ I HAVE IDENTIFIED TWO DAY SECTIONS OF THE SAME 
CLASS. 

□ THESE CLASSES ARE AND THEY MEET AT: 



□ I WILL USE CLASSROOM ASSESSMENTS IN MY 
CLASS. 

□ I WILL NQI USE CLASSROOM ASSESSMENTS IN MY 
CLASS. 

□ TODAY, , 1993, WAS THE FIRST 

MEETING OF MY FEEDBACK SECTION CLASS. I USED 
CLASSROOM ASSESSMENT TECHNIQUES TO SET THE 
TONE AND ASSESS MY STUDENTS. 

□ TODAY, , 1993, WAS THE FIRST 

MEETING OF MY NQN-FEEDBACK CLASS. I TAUGHT 
HOW I USED TO TEACH BEFORE LEARNING ABOUT 
CLASSROOM ASSESSMENT TECHNIQUES. 

□ I AM CLEAR THAT I AM TO USE FREQUENT 
ASSESSMENTS IN THE FEEDBACK CLASS, AND NONE 
AT ALL IN THE MQN-FEEDBACK CLASS. 

□ TWO WEEKS HAVE PASSED, AND I HAVE USED AT 
LEAST ONE NEW ASSESSMENT IN MY FEEDBACK 
CLASS. 

□ FIRST CENSUS LISTS HAVE COME OUT AND I MADE A 
COPY OF MINE IN BOTH CLASSES FOR THE 
RESEARCHERS. 

□ TWO WEEKS HAVE PASSED, AND I HAVE USED AT 
LEAST ONE NEW ASSESSMENT IN MY FEEDBACK 
CLASS. 

□ TWO WEEKS HAVE PASSED, AND I HAVE USED AT 
LEAST 1 NEW ASSESSMENT IN MY FEEDBACK CLASS. 



□ TWO WEEKS HAVE PASSED, AND I HAVE USED AT 
LEAST ONE NEW ASSESSMENT IN MY FEEDBACK 
CLASS. 

□ IT IS TIME FOR THE CECUl INSTRUMENT TO BE 
ADMINISTERED IN BOTH MY CLASSES. THE 
RESEARCHER SHOULD PLAN TO COLLECT DATA IN MY 
FEEDBACK CLASS ON AT 



□ THE RESEARCHER SHOULD PLAN TO COME TO MY 

NON-FEEDBACK CLASS ON AT 



□ I HAVE TOLD MY STUDENTS THAT THE RESEARCHER IS 
COMING IN AND ALLOWED TIME IN MY CLASS 
SCHEDULES. 

□ TWO \A(EEKS HAVE PASSED, AND I HAVE USED AT 
LEAST ONE NEW ASSESSMENT IN MY FEEDBACK 
CLASS. 

□ TWO WEEKS HAVE PASSED, AND I HAVE USED AT 
LEAST ONE NEW ASSESSMENT IN MY FEEDBACK 
CLASS. 

□ TWO WEEKS HAVE PASSED, AND I HAVE USED AT 
LEAST ONE NEW ASSESSMENT IN MY FEEDBACK 
CLASS. 

□ I HAVE NOW COMPLETED THE SEMESTER (OR 
QUARTER) AND HAVE CLEARLY TREATED MY TWO 
POPULATIONS DIFFERENTLY WHILE TEACHING THE 
SAME COURSE CONTENT. 

□ IF I HAVE ANY QUESTIONS, I CAN CALL MICHELLE 
KALINA AT 916-624-3333 EXT. 2274 OR ANITA CATLIN 
AT 707-253-3135. 

□ I KNOW THAT I AM A VERY VALUABLE PERSON AND 
HAVE CONTRIBUTED TREMENDOUSLY TO THE SUCCESS 
OF THIS PROJECT. THE CHANCELLOR'S OFFICE, MY 
COLLEGE, ANITA AND MICHELLE THANK ME FROM THE 
BOTTOM OF THEIR COLLECTIVE HEARTS. 
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INSTRUCTOR SURVEY FOR MATCHED CLASS STUDY FALL, 1992 

Directions: If your students exhibited tlie beliavior, place a check mark in the 
appropriate box. 



BEHAVIOR: 

Did most of your students: 

Come to class on time? 

Attend Regularly? 

Come with their textbooks? 

Come with paper to write on? 

Come with pens/pencils? 

Come with homework done? 

Come late to class? 

Come without textbooks? 

Come without paper? 

Come without pens/pencils? 

Come without homework done? 

Remain quiet while teacher was 
talking? 

Remain quite while another 
student was talking? 

Ask questions about classroom 
material? 

Ask irrelevant questions? 

Talk regardless of who else was 
talking? 

Increased student use of office 
hours? 

COMMENTS: §2 
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NON-FEEDBACK FEEDBACK 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 



□ 



November 5, 19 92 



TO: CAP Faculty Research Participants, spring '93 Matched 

Class study 

FROM: Michelle Kalina, Research Director 
Anita Catlin, pii Project Director 

RE: Your participation in the Pll #92-0016 Grant study 

spring '93 



Thank you all for agreeing to participate in the classroom 
Assessment project (CAP) study which will occur in the Sprina 
semester, 1993. This is a study to examine the effects of clasSroom 
assessment techniques on student outcomes. Following is a list of 
stud"^* needs your agreement in order to participate in this 

1. Each of you are teaching two daytime sections of the same 
class in the Spring semester. 

2. You will designate one section a NO FEEDBACK fNF) section 
^llJ^^v absolutely no Classroom Assessment Techniques 
(CATS) with that class for the entire semester. 

3. You will designate the other section a FEEDBACK IF\ section 
and will use frequent CATs with this section. Frequent 
means as many as possible, as often as possible. A minimum 
of one CAT IS required every two weeks but we encourage vo u 
to use C ATS as often as posaibia. you will use CATs 

with this group for the entire semester. 

4. Each of you will schedule approximately 20 minutes of class 
time, during the tenth week of the semester, in each class 
involved in the study, so the CUCEI Inventory may be 
administered. You should establish the time and date with 
your researcher now. The Inventory is done anonymously, as 
are all CATs. Please inform your classes that someone will 
be coming in to do the survey. 

5. If your college does not store and keep first census 
student lists, please make certain that you keep your copy. 

Some demographic data will be collected relevant to the students 
participating in the study. However, the CUCEI Inventory win So? 
'! V'lT'.t'' individual or by school. Analysis will be by grS5| 
ti'iV ho "° feedback group). No one person or scSoo? 

nil] • ^^^''^'■^^^^ and all materials will be shredded after 
analysis has occurred. xouueu aiter 

If you have any questions and/or problems, please contact the 

c:?lirnS7)*"2/3°"3A'/"^'°j; - <=-ll eftSeJ^AniJ: 

catlin (707) 253-3135 of Michelle Kalina (916) 624-3333 X2274. 
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Please sign, detach and return this page to your CAP Researcher* 
Ycur signature indicates that you have read and understand the 
criteria necessary for maintaining the study's integrity and you 
agree to all the terms. 



Signature 



Print your name 



School 



Mail this to Anita Catlin, Napa Valley College, 2277 Napa-Vallejo 
Highway, Napa, CA 94558. 
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WHAT THE CATS RESEARCH SAYS TO US ABOUT 
WOMEN'S EDUCATION 

by Lauren Coodley, Professor 
Napa Valley College 

The results of this reseach, which indicate "a very strong connection between the use 
of CATs and a positive learning experience for women... even more strongly for new 
majority women" offer fascinating validation of recent theoretical developments in the 
study of women and education. 

Specifically, these research results can be analyzed from the perspective of the AAUW 
Report, How Schools Shortchange Girls :^ from the work of Belenky, et.al., in 
Women's Ways of Knowing :^ from the leading edge pedagogy which demystifies math 
for girls at the Equals Program, Lawrence Hall of Science;^ and from the experiments 
in feminist education which have been quietly undertaken by thousands of instructors 
over the past few decades, discussed in such books as Theories of Women's Studies / 

As well as teaching women to overcome math anxiety for many years, I have also had 
the opportunity to teach the history of American women. 1 teach about Anne 
Hutchinson, the first American "women rebel," who was expelled from the church in 
1636 for teaching women they could achieve salvation without church authorities. 
Anne began to preach the Antinomian philosophy which, reduced the powers of the 
ministers as intermediaries to God, and empowered the people, women as well as 
men, to develop a state of grace in direct communication with their Deity. ^ 

Inadvertently or not, the developers of CATs have handed the nonvocal, well behaved, 
self conscious women students of today (especially those at community college, who 
have usually not experienced outstanding academic success) a tool that similarly 
empowers them and changes the shape of their educational experience. 

From the AAUW Research Report, we have dramatic evidence that girls are 
overlooked in K-12 classrooms; that overwhelmingly, teacher attention goes to more 
vocal and demanding male students.® We know that women do not continue in 
advanced math and science classes past high school, nor do they imagine themselves 
becoming scientists and engineers-even those who G^cceed in these classes.^ The 
consequences of the feminization of poverty, the inability of women to independently 
support their families, are well known. Thus, the price to young women of being 
docile, submissive and silent is high in the traditional classroom. 

In Women's Wavs of Knowing . Patricia Palmieri suggests, that "a better understanding 
of women's experience would permit, even force, a far-reaching revision of the 
broader fields of higher education and intellectual life in the United States."® Many 
women come to college uncertain of their abilities; "our interviews have convinced us 
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that every woman, regardless of age, class, ethnicity, and academic achievement, 
needs to know that she is capable of intelligent thought, and she needs to know it 
right away."^ 

Peter Elbow, quoted in Women^s Wavs of Knowing , comments that traditional 
professors... "stand up in front of students and say. ..get what is inside me inside you; 
listen to me; be like me; I am important." In contrast, he advocates competency- 
based programs where the teacher is more a "coach or an ally," where the teachers 
attempts to confirm the student as a knower.""^ The authors argue that "evaluation 
in the connected mode requires that the standards of evaluation be constructed in 
collaboration with students. "^^ 

The book goes on to propose a model of "the teacher as midwife" (rather than 
banker), who "assist the students in giving birth to their own ideas, in making their 
own tacit knowledge explicit;" who "focus not on their own knowledge, but on the 
students knowledge, "^^ and who "receives and accepts the students feeling toward 
the subject matter. "^^ 

Women have been particularly intimidated by and unsuccessful in math and science 
classes taught in a hierarchical, competitive, and isolated manner. Educators at the 
Lawrence Hall of Science, inspired by Sheila Tobias' blistering critique of traditional 
math instruction,^"^ have pioneered cooperative group instruction in math, the use of 
journals, alternative assessment methods and real-world word problems to address 
this situation. ""^ Although many secondary and college math instructors are not 
interested in these approaches, CATs enables formerly silent women students to "give 
voice" to their concerns and suggestions. 

Educators influenced by feminist pedagogy have wrestled for several decades with the 
question of authority in the classroom, of student empowerment, of democratic 
structure which does not become the tyranny of the vocal minority. In CATs we have 
a system which is scrupulously fair and evenhanded, which mandates student input 
and yet leaves the instructor with the choices of when and how to utilize the 
information received. In a society where adult authority is still overwhelmingly male, 
where our educational institutions were designed by men, and where women are 
struggling on a daily basis with patriarchal power by fathers and husbands, there has 
been, invisibly, a "Chilly College Climate for Women. "^^ 

Although no assumptions were made about the effect of CATs on women students, 
although there was no specific hypothesis about the effect of CATs by gender, every 
instructor knows that most women are too shy to volunteer information or ask 
questions in a large classroom. By making everybody equal, by giving everybody a 
voice, the developers and users of CATs have levelled the playing field (to use a 
typically masculinist analogy). Small wonder that so many women walked out of 
church with Anne Hutchinson when she was told "You have rather bine a Magistral- 
than a Subject.. ."^^ With CATs, especially in the drastically underrepresented ..dc' 
well-paid science and math fields, women students enjoyed their new sense of 
entitlement in the classroom environment, according to the data represented in this 
study. 
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"Connected learning," as Belenky et. al., have titled it/® is effective with both men 
and wonnen of all ethnicities. The results of this study should remind us that all things 
have not been equal in the college classroom, but that they can be equalized, with 
exciting potential for the development of the unused potential of our previously 
silenced students. 



^ American Association of University Women, (1 992) How Schools Shortchange 
Girls , AAUW Educational Foundation, 1111 16th Street N.W., Washington, D.C. 

^Belenky, Clinchy, Goldberg, Tarule. (1986). Women's Ways of Knowing , Basic 
Books, New York. 

^Down'e, Slesnick and Stenmark. (1981). Math for Girls and Other Problem 
Solvers , Berkeley: Regents of University of California. 

"^Bowles, G. and Duelle-Klein, R. (1983). Theories of Women's Studies , London: 
Routledge and Kegan Paul. 

^Koehler, L. (1 973). The case of the American Jezebels: Anne Hutchinson and 
female agitation during the years of Antinomian Turmoil 1 636-1 640, in Our American 
Sisters , Boston: Allyn and Bacon. 

®AAUW Report, op cit. Part 4, Chapter 2. 

^AAUW Report, op cit. Part 2, Chapter 2. 

®Belenky, op cit, pp. 191 . 

^Belenky, op cit, pp. 193. 

^^Belenky, op cit, pp. 208. 

^^BelenKy, op cit, pp. 209. 

^^Belenky, op cit, pp. 217. 

^^Belenky, op cit, pp. 217. 

^"^Tobias, S. (1978). Overcoming Math Anxiety , Boston: Houghton Mifflin. 

^^Lawrence Hall of Science publications list, UC, Berkeley, Berkeley, CA 94728. 

^^Hall, R. (1984) A Chilly Classroom Climate for Women , Project on the Status 
and Education of Women, Association of American Colleges. 

^^Koehler, L. (1973). The case of the American Jezebels: Anne Hutchinson and 
female agitation during the years of Antinomian Turmoil 1 636-1 640, in Our American 
Sisters , Boston: Allyn and Bacon, pp. 58. 
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